Section: New Software and Platforms

CodaLab

Keywords Benchmarking, competitions.

Functional Description

Challenges in machine learning and data science are competitions running over several weeks or months to resolve problems using provided datasets or simulated environments. Challenges can be thought of as crowdsourcing, benchmarking, and communication tools. They have been used for decades to test and compare competing solutions in machine learning in a fair and controlled way, to eliminate “inventor-evaluator" bias, and to stimulate the scientific community while promoting reproducible science.

Codalab Competitions (http://competitions.codalab.org) is a project that was started by Microsoft Research in 2013 in which Isabelle Guyon has taken an active part, to promote the use of challenges in Machine Learning and Data Science. The TAO team has been selected to take over the project under Isabelle Guyon's leadership. The transfer has been successfully completed in the fall 2016. New features are being implemented, including developing a Wizard http://staging.chalab.eu/.

With already over 50 public competitions (including this year the Data Science Game, a student Olympiad co-organized by our PhD. student Benjamin Donnot http://www.datasciencegame.com/, the AutoML challenge http://automl.chalearn.org/ [42] and a new contest in the LAP challenge series http://chalearnlap.cvc.uab.es/ [47], co-organized by Isabelle Guyon), Codalab is taking momentum in medical imaging, computer vision, time series prediction, text mining, and other applications. TAO is going to continue expanding Codalab to accommodate new needs. For example, two competitions in preparation – TrackML competition (in High Energy Physics) [72] and the See.4C competition (spatio-temporal time series in collaboration with RTE) [48] – will require code submission, permitting to benchmark methods in a controlled environment. We are re-designing the backend of CodaLab to allow organizers to add more servers to satisfy on-the-fly demands of new competitions. Other features coming soon will be the possibility of interacting with a data generating model (rather than analyzing “canned” data), which enables the organization of reinforcement learning competitions and the possibility of organizing “coopetitions” (a mix of competition and collaboration). Other existing challenge platforms are too restrictive to simulate collaboration between participants and implement “coopetitions”. Our starting PhD. student Lisheng Sun designed and implemented a first prototype of coopetition “Beat AutoSKLearn", which was run at the NIPS Challenges in Machine Learning workshop (CiML 2016 http://ciml.chalearn.org/).

• Contact: Isabelle Guyon

• URL: http://competitions.codalab.org